Repository navigation
feat: replicated schema migration - #3309
SpencerTorres wants to merge 6 commits into
Conversation
🦋 Changeset detectedLatest commit: ca2e4a3 The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 2 Skipped Deployments
|
🔴 Tier 4 — CriticalTouches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI. Why this tier:
Review process: Deep review from a domain expert. Synchronous walkthrough may be required. Stats
|
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
|
|
||
| // A fallback database keeps the plain engines it was seeded with. | ||
| replicated := false | ||
| if database == cfg.Database { |
There was a problem hiding this comment.
🔵 minor — Replicated mode is dropped without a warning when the fallback database is used
With HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED=true and a fallback that wins (the target is missing and the fallback has tables), planReplicated never runs. The fallback is seeded with plain engines and nothing says Replicated was not applied. Log a warning in that branch, for example: if cfg.Replicated && database != cfg.Database { log.Printf("WARNING: Replicated engines not applied; staying on fallback %s", database) }.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
PR Review
3 finding(s): 🔴 0 P0/P1 · 🟡 0 P2 · 🔵 3 P3 No inline comments. 3 P3 folded at the bottom. 🔵 3 P3 — no action needed before merge
Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides. |
|
| // --------------------------------------------------------------------------- | ||
| // replicated engines | ||
| // --------------------------------------------------------------------------- |
There was a problem hiding this comment.
The new section dividers violate the AGENTS.md directive: “No banner comments, no ASCII section dividers.” This rule also applies to tests. Remove this banner and the matching fallback banner at lines 974–976 before merging to satisfy the repository requirement.
Context Used: AGENTS.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Deep ReviewScope: PR #3309 — opt-in ClickHouse Intent: This is a carefully written, well-tested change. The resolved-database handoff is robust: ✅ No critical issues found. 🟡 P2 -- recommended
🔵 P3 nitpicks (3)
Reviewers (9 dispatched): correctness, data-migrations, reliability, adversarial, testing, maintainability, project-standards, security, previous-comments. Note on coverage: the parallel reviewer sub-agents had not returned at the point synthesis was required, so these findings reflect the orchestrator's own verified read of the diff rather than a merge of the sub-agent outputs. Severities are graded conservatively; re-running the review will add corroboration and may surface additional lower-confidence items. Testing gaps: No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
E2E Test Results✅ All tests passed • 474 passed • 1 skipped • 1251s
Tests ran across 4 shards in parallel. |
wrn14897
left a comment
There was a problem hiding this comment.
Could you elaborate on why we need HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASE?
Seed hyperdx_sessions_compat (new on main) with ${ENGINE_PREFIX}MergeTree,
and set the text index block size on the resolved database rather than the
configured one.
| -- +goose Up | ||
| -- +goose NO TRANSACTION | ||
| CREATE DATABASE IF NOT EXISTS ${DATABASE}; | ||
| CREATE DATABASE IF NOT EXISTS ${DATABASE}${DATABASE_ENGINE}; |
There was a problem hiding this comment.
🟡 P2 — Replicated database is only created on the node migrate connects to
A Replicated database has to be created on every replica, either with ON CLUSTER or by running CREATE DATABASE on each node. Migrate runs a plain CREATE DATABASE IF NOT EXISTS over the single CLICKHOUSE_ENDPOINT. If collectors reach ClickHouse through a load balancer or one shared host, only that one node joins the database. Inserts routed to any other node then fail because the database does not exist there. Either document that the database must already exist on every node or be created by connecting to each node, or accept an optional cluster name and emit ON CLUSTER '<cluster>'. Add this to the changeset and to the docker-compose.dev.yml comment as well.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| tempDir, err := processSchemaDir(cfg.SchemaDir, cfg.Database, ttlExprs) | ||
| // The legacy tables check is best effort unless the fallback depends on it. | ||
| var legacyHasTables bool | ||
| if legacy := legacyDatabase(cfg.FallbackDatabase); legacy != cfg.Database { |
There was a problem hiding this comment.
🟡 P2 — New legacy-tables warning fires on every boot, including with both flags unset
hasClickStackTables(default) now runs whenever the target is not default. When neither flag is set, dbExists is never queried, so resolveDatabase can't tell a deliberate past move from a fresh one. Any deployment with tables in both default and its target logs "not migrated ... HyperDX sources created earlier still point at default" on every start, with no way to silence it. This includes the smoke-test databases on a shared server. The same thing happens forever once a fallback user moves to the target. This contradicts "nothing changes when unset". Fix options: gate the warning on the target not existing (always call getDatabaseEngine), or on the target not yet having ClickStack tables, or show it only on first seed.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| log.Printf("WARNING: %s", warning) | ||
| } | ||
|
|
||
| // A fallback database keeps the plain engines it was seeded with. |
There was a problem hiding this comment.
🟡 P2 — REPLICATED=true is silently ignored when the fallback database is chosen
When resolveDatabase returns the fallback, planReplicated is skipped. An operator who set HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED=true gets only the fallback warning, not the Replicated-specific one. They also get plain engines, even if the fallback database is itself Replicated. Fix: check the fallback's engine with getDatabaseEngine and pass it to planReplicated, or log a warning when cfg.Replicated is set and the fallback wins.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
…engine, document per-node Replicated setup
…spencer/replicated-engine
Replicated can only be set when a database is created, and our new helm changes use
Usually empty, but when replicated is enabled we set it as replicated engine
For the purpose of checking for an existing install yes |
| } | ||
|
|
||
| // A fallback already exists, so only its own engine can make it Replicated. | ||
| if cfg.Replicated && database != cfg.Database { |
There was a problem hiding this comment.
🟡 P2 — With both REPLICATED and FALLBACK_DATABASE set, the Replicated warning gives advice the operator has already followed
In this case resolveDatabase has moved the seed to the fallback database, so planReplicated tells the operator to "set HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE to a new database name". They already did that. The fallback is what blocks replication. When database != cfg.Database, use a warning that tells them to clear HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASE (or to replicate the fallback database), for example by passing the fallback context into planReplicated.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Summary
Adds opt-in support for ClickHouse's Replicated engines to the schema migration, so the database and tables can replicate across nodes.
HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED=truecreates the database with theReplicatedengine and seedsReplicated*tables. An existing non-Replicated database keeps plain engines, with a warning.HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASEkeeps writing to an existing database when the configured one doesn't exist yet. The collector is pointed at whichever database the migration used.Both are off by default, and nothing changes when unset. Tested with unit tests and against ClickHouse 25.7 and 26.2 with Keeper, including multi-node replication.
References